On the Mathematical Foundations of Learning Felipe Cucker and Steve Smale

approximation error A linear operator L H H on a Hilbert space H is said to be self adjoint if for all f g H hLf gi hf Lgi It is said to be positive resp strictly positive if it is self adjoint and for all non trivial f H hLf fi resp hLf fi The next result the Spectral Theorem for compact operators see Section of for a proof will be useful in this and the next chapter Theorem Let L be a compact linear operator on an in nite dimensional Hilbert space H Then there exists in H a complete orthonormal system f g con sisting of the eigenvectors of L If k is the eigenvalue corresponding to k then the set f kg is either nite or k when k In addition maxk j kj kLk The eigenvalues are real if L is self adjoint If in addition L is positive then k for all k and if L is strictly positive then k for all k If L is a strictly positive operator then L is de ned for all by L X ak k X kak k If L is de ned by the same formula on the subspace S nX ak k j X ak k is convergent o For the expression kL ak must be understood as if a S Theorems and in this and the next section are taken from where one can nd a more substantial development of the approximation error Theorem Let H be a Hilbert space and A a self adjoint strictly positive compact operator on H Let s r R such that s r Let Then for all a H min b H kb ak kA sbk rkA srak Let R Then for all a H min b s t kA sbk R kb ak R r s r kA rak s s r In both cases the minimizer !b exists and is unique In addition in part !b Id A s a Proof First note that by replacing A by A we can reduce the problem in both parts and to the case s Now for part consider b kb ak kA bk If a point !b minimizes then it must be a zero of the derivative D That is !b satis es Id A !b a which implies !b Id A a Note that the operator Id A is invertible since it is the sum of the identity and a positive operator FELIPE CUCKER AND STEVE SMALE If denotes the eigenvalues of A !b k Id A Id ak kA Id A ak X